{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convert Dataset Formats\n",
    "\n",
    "This recipe demonstrates how to use FiftyOne to convert datasets on disk between common formats."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Requirements\n",
    "\n",
    "This notebook contains bash commands. To run it as a notebook, you must install the [Jupyter bash kernel](https://github.com/takluyver/bash_kernel) via the command below.\n",
    "\n",
    "Alternatively, you can just copy + paste the code blocks into your shell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "pip install bash_kernel\n",
    "python -m bash_kernel.install"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this recipe we'll use the [FiftyOne Dataset Zoo](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/zoo_datasets.html) to download some open source datasets to work with.\n",
    "\n",
    "Specifically, we'll need [TensorFlow](https://www.tensorflow.org/) and [TensorFlow Datasets](https://www.tensorflow.org/datasets) installed to [access the datasets](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/zoo_datasets.html#customizing-your-ml-backend):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# See https://www.tensorflow.org/install for more install options\n",
    "pip install tensorflow\n",
    "pip install tensorflow-datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download datasets\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the test split of the [CIFAR-10 dataset](https://www.cs.toronto.edu/~kriz/cifar.html) from the [FiftyOne Dataset Zoo](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/zoo_datasets.html) using the command below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading split 'test' to '~/fiftyone/cifar10/test'\n",
      "Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ~/fiftyone/cifar10/tmp-download/cifar-10-python.tar.gz\n",
      "170500096it [00:04, 35887670.65it/s]                                            \n",
      "Extracting ~/fiftyone/cifar10/tmp-download/cifar-10-python.tar.gz to ~/fiftyone/cifar10/tmp-download\n",
      " 100% |███| 10000/10000 [5.2s elapsed, 0s remaining, 1.8K samples/s]      \n",
      "Dataset info written to '~/fiftyone/cifar10/info.json'\n"
     ]
    }
   ],
   "source": [
    "# Download the test split of CIFAR-10\n",
    "fiftyone zoo download cifar10 --split test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the validation split of the [KITTI dataset]( http://www.cvlibs.net/datasets/kitti) from the [FiftyOne Dataset Zoo](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/zoo_datasets.html) using the command below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Split 'validation' already downloaded\n"
     ]
    }
   ],
   "source": [
    "# Download the validation split of KITTI\n",
    "fiftyone zoo download kitti --split validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The fiftyone convert command"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [FiftyOne CLI](https://voxel51.com/docs/fiftyone/cli/index.html) provides a number of utilities for importing and exporting datasets in a variety of common (or custom) formats.\n",
    "\n",
    "Specifically, the `fiftyone convert` command provides a convenient way to convert datasets on disk between formats by specifying the [fiftyone.types.Dataset](https://voxel51.com/docs/fiftyone/api/fiftyone.types.html#fiftyone.types.dataset_types.Dataset) type of the input and desired output.\n",
    "\n",
    "FiftyOne provides a collection of [builtin types](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#supported-formats) that you can use to read/write datasets in common formats out-of-the-box:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"convert-recipes-table\">\n",
    "\n",
    "| Dataset format                                                                                                                                       | Import Supported? | Export Supported? | Conversion Supported? |\n",
    "| ---------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | ----------------- | --------------------- |\n",
    "| [ImageDirectory](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#imagedirectory)                                         | ✓                 | ✓                 | ✓                     |\n",
    "| [VideoDirectory](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#videodirectory)                                         | ✓                 | ✓                 | ✓                     |\n",
    "| [FiftyOneImageClassificationDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#fiftyoneimageclassificationdataset) | ✓                 | ✓                 | ✓                     |\n",
    "| [ImageClassificationDirectoryTree](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#imageclassificationdirectorytree)     | ✓                 | ✓                 | ✓                     |\n",
    "| [TFImageClassificationDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#tfimageclassificationdataset)             | ✓                 | ✓                 | ✓                     |\n",
    "| [FiftyOneImageDetectionDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#fiftyoneimagedetectiondataset)           | ✓                 | ✓                 | ✓                     |\n",
    "| [COCODetectionDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#cocodetectiondataset)                             | ✓                 | ✓                 | ✓                     |\n",
    "| [VOCDetectionDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#vocdetectiondataset)                               | ✓                 | ✓                 | ✓                     |\n",
    "| [KITTIDetectionDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#kittidetectiondataset)                           | ✓                 | ✓                 | ✓                     |\n",
    "| [YOLODataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#yolodataset)                                               | ✓                 | ✓                 | ✓                     |\n",
    "| [TFObjectDetectionDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#tfobjectdetectiondataset)                     | ✓                 | ✓                 | ✓                     |\n",
    "| [CVATImageDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#cvatimagedataset)                                     | ✓                 | ✓                 | ✓                     |\n",
    "| [CVATVideoDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#cvatvideodataset)                                     | ✓                 | ✓                 | ✓                     |\n",
    "| [FiftyOneImageLabelsDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#fiftyoneimagelabelsdataset)                 | ✓                 | ✓                 | ✓                     |\n",
    "| [FiftyOneVideoLabelsDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#fiftyonevideolabelsdataset)                 | ✓                 | ✓                 | ✓                     |\n",
    "| [BDDDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#bdddataset)                                                 | ✓                 | ✓                 | ✓                     |\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition, you can define your own [custom dataset types](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#custom-formats) to read/write datasets in your own formats.\n",
    "\n",
    "The usage of the `fiftyone convert` command is as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "usage: fiftyone convert [-h] [--input-dir INPUT_DIR] [--input-type INPUT_TYPE]\n",
      "                        [--output-dir OUTPUT_DIR] [--output-type OUTPUT_TYPE]\n",
      "\n",
      "Convert datasets on disk between supported formats.\n",
      "\n",
      "    Examples::\n",
      "\n",
      "        # Convert an image classification directory tree to TFRecords format\n",
      "        fiftyone convert \\\n",
      "            --input-dir /path/to/image-classification-directory-tree \\\n",
      "            --input-type fiftyone.types.ImageClassificationDirectoryTree \\\n",
      "            --output-dir /path/for/tf-image-classification-dataset \\\n",
      "            --output-type fiftyone.types.TFImageClassificationDataset\n",
      "\n",
      "        # Convert a COCO detection dataset to CVAT image format\n",
      "        fiftyone convert \\\n",
      "            --input-dir /path/to/coco-detection-dataset \\\n",
      "            --input-type fiftyone.types.COCODetectionDataset \\\n",
      "            --output-dir /path/for/cvat-image-dataset \\\n",
      "            --output-type fiftyone.types.CVATImageDataset\n",
      "\n",
      "optional arguments:\n",
      "  -h, --help            show this help message and exit\n",
      "  --input-dir INPUT_DIR\n",
      "                        the directory containing the dataset\n",
      "  --input-type INPUT_TYPE\n",
      "                        the fiftyone.types.Dataset type of the input dataset\n",
      "  --output-dir OUTPUT_DIR\n",
      "                        the directory to which to write the output dataset\n",
      "  --output-type OUTPUT_TYPE\n",
      "                        the fiftyone.types.Dataset type to output\n"
     ]
    }
   ],
   "source": [
    "fiftyone convert -h"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convert CIFAR-10 dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you downloaded the test split of the CIFAR-10 dataset above, it was written to disk as a dataset in [fiftyone.types.FiftyOneImageClassificationDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#fiftyoneimageclassificationdataset) format.\n",
    "\n",
    "You can verify this by printing information about the downloaded dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***** Dataset description *****\n",
      "The CIFAR-10 dataset consists of 60000 32 x 32 color images in 10\n",
      "    classes, with 6000 images per class. There are 50000 training images and\n",
      "    10000 test images.\n",
      "\n",
      "    Dataset size:\n",
      "        132.40 MiB\n",
      "\n",
      "    Source:\n",
      "        https://www.cs.toronto.edu/~kriz/cifar.html\n",
      "    \n",
      "***** Supported splits *****\n",
      "test, train\n",
      "\n",
      "***** Dataset location *****\n",
      "~/fiftyone/cifar10\n",
      "\n",
      "***** Dataset info *****\n",
      "{\n",
      "    \"name\": \"cifar10\",\n",
      "    \"zoo_dataset\": \"fiftyone.zoo.torch.CIFAR10Dataset\",\n",
      "    \"dataset_type\": \"fiftyone.types.dataset_types.FiftyOneImageClassificationDataset\",\n",
      "    \"num_samples\": 10000,\n",
      "    \"downloaded_splits\": {\n",
      "        \"test\": {\n",
      "            \"split\": \"test\",\n",
      "            \"num_samples\": 10000\n",
      "        }\n",
      "    },\n",
      "    \"classes\": [\n",
      "        \"airplane\",\n",
      "        \"automobile\",\n",
      "        \"bird\",\n",
      "        \"cat\",\n",
      "        \"deer\",\n",
      "        \"dog\",\n",
      "        \"frog\",\n",
      "        \"horse\",\n",
      "        \"ship\",\n",
      "        \"truck\"\n",
      "    ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "fiftyone zoo info cifar10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The snippet below uses `fiftyone convert` to convert the test split of the CIFAR-10 dataset to [fiftyone.types.ImageClassificationDirectoryTree](https://voxel51.com/docs/fiftyone/user_guide/export_datasets.html#imageclassificationdirectorytree) format, which stores classification datasets on disk in a directory tree structure with images organized per-class:\n",
    "\n",
    "```\n",
    "<dataset_dir>\n",
    "├── <classA>/\n",
    "│   ├── <image1>.<ext>\n",
    "│   ├── <image2>.<ext>\n",
    "│   └── ...\n",
    "├── <classB>/\n",
    "│   ├── <image1>.<ext>\n",
    "│   ├── <image2>.<ext>\n",
    "│   └── ...\n",
    "└── ...\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading dataset from '~/fiftyone/cifar10/test'\n",
      "Input format 'fiftyone.types.dataset_types.FiftyOneImageClassificationDataset'\n",
      " 100% |███| 10000/10000 [4.2s elapsed, 0s remaining, 2.4K samples/s]      \n",
      "Import complete\n",
      "Exporting dataset to '/tmp/fiftyone/cifar10-dir-tree'\n",
      "Export format 'fiftyone.types.dataset_types.ImageClassificationDirectoryTree'\n",
      " 100% |███| 10000/10000 [6.2s elapsed, 0s remaining, 1.7K samples/s]        \n",
      "Export complete\n"
     ]
    }
   ],
   "source": [
    "INPUT_DIR=$(fiftyone zoo find cifar10 --split test)\n",
    "OUTPUT_DIR=/tmp/fiftyone/cifar10-dir-tree\n",
    "\n",
    "fiftyone convert \\\n",
    "    --input-dir ${INPUT_DIR} --input-type fiftyone.types.FiftyOneImageClassificationDataset \\\n",
    "    --output-dir ${OUTPUT_DIR} --output-type fiftyone.types.ImageClassificationDirectoryTree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's verify that the conversion happened as expected:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 0\n",
      "drwxr-xr-x    12 voxel51  wheel   384B Jul 14 11:08 .\n",
      "drwxr-xr-x     3 voxel51  wheel    96B Jul 14 11:08 ..\n",
      "drwxr-xr-x  1002 voxel51  wheel    31K Jul 14 11:08 airplane\n",
      "drwxr-xr-x  1002 voxel51  wheel    31K Jul 14 11:08 automobile\n",
      "drwxr-xr-x  1002 voxel51  wheel    31K Jul 14 11:08 bird\n",
      "drwxr-xr-x  1002 voxel51  wheel    31K Jul 14 11:08 cat\n",
      "drwxr-xr-x  1002 voxel51  wheel    31K Jul 14 11:08 deer\n",
      "drwxr-xr-x  1002 voxel51  wheel    31K Jul 14 11:08 dog\n",
      "drwxr-xr-x  1002 voxel51  wheel    31K Jul 14 11:08 frog\n",
      "drwxr-xr-x  1002 voxel51  wheel    31K Jul 14 11:08 horse\n",
      "drwxr-xr-x  1002 voxel51  wheel    31K Jul 14 11:08 ship\n",
      "drwxr-xr-x  1002 voxel51  wheel    31K Jul 14 11:08 truck\n"
     ]
    }
   ],
   "source": [
    "ls -lah /tmp/fiftyone/cifar10-dir-tree/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 8000\n",
      "drwxr-xr-x  1002 voxel51  wheel    31K Jul 14 11:08 .\n",
      "drwxr-xr-x    12 voxel51  wheel   384B Jul 14 11:08 ..\n",
      "-rw-r--r--     1 voxel51  wheel   1.2K Jul 14 11:23 000004.jpg\n",
      "-rw-r--r--     1 voxel51  wheel   1.1K Jul 14 11:23 000011.jpg\n",
      "-rw-r--r--     1 voxel51  wheel   1.1K Jul 14 11:23 000022.jpg\n",
      "-rw-r--r--     1 voxel51  wheel   1.3K Jul 14 11:23 000028.jpg\n",
      "-rw-r--r--     1 voxel51  wheel   1.2K Jul 14 11:23 000045.jpg\n",
      "-rw-r--r--     1 voxel51  wheel   1.2K Jul 14 11:23 000053.jpg\n",
      "-rw-r--r--     1 voxel51  wheel   1.3K Jul 14 11:23 000075.jpg\n"
     ]
    }
   ],
   "source": [
    "ls -lah /tmp/fiftyone/cifar10-dir-tree/airplane/ | head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's convert the classification directory tree to [TFRecords](https://voxel51.com/docs/fiftyone/user_guide/export_datasets.html#tfimageclassificationdataset) format!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading dataset from '/tmp/fiftyone/cifar10-dir-tree'\n",
      "Input format 'fiftyone.types.dataset_types.ImageClassificationDirectoryTree'\n",
      " 100% |███| 10000/10000 [4.0s elapsed, 0s remaining, 2.5K samples/s]      \n",
      "Import complete\n",
      "Exporting dataset to '/tmp/fiftyone/cifar10-tfrecords'\n",
      "Export format 'fiftyone.types.dataset_types.TFImageClassificationDataset'\n",
      "   0% ||--|     1/10000 [23.2ms elapsed, 3.9m remaining, 43.2 samples/s] 2020-07-14 11:24:15.187387: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\n",
      "2020-07-14 11:24:15.201384: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f83df428f60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n",
      "2020-07-14 11:24:15.201405: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version\n",
      " 100% |███| 10000/10000 [8.2s elapsed, 0s remaining, 1.3K samples/s]        \n",
      "Export complete\n"
     ]
    }
   ],
   "source": [
    "INPUT_DIR=/tmp/fiftyone/cifar10-dir-tree\n",
    "OUTPUT_DIR=/tmp/fiftyone/cifar10-tfrecords\n",
    "\n",
    "fiftyone convert \\\n",
    "    --input-dir ${INPUT_DIR} --input-type fiftyone.types.ImageClassificationDirectoryTree \\\n",
    "    --output-dir ${OUTPUT_DIR} --output-type fiftyone.types.TFImageClassificationDataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's verify that the conversion happened as expected:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 29696\n",
      "drwxr-xr-x  3 voxel51  wheel    96B Jul 14 11:24 .\n",
      "drwxr-xr-x  4 voxel51  wheel   128B Jul 14 11:24 ..\n",
      "-rw-r--r--  1 voxel51  wheel    14M Jul 14 11:24 tf.records\n"
     ]
    }
   ],
   "source": [
    "ls -lah /tmp/fiftyone/cifar10-tfrecords"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convert KITTI dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you downloaded the validation split of the KITTI dataset above, it was written to disk as a dataset in [fiftyone.types.FiftyOneImageDetectionDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#fiftyoneimagedetectiondataset) format.\n",
    "\n",
    "You can verify this by printing information about the downloaded dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***** Dataset description *****\n",
      "KITTI contains a suite of vision tasks built using an autonomous\n",
      "    driving platform.\n",
      "\n",
      "    The full benchmark contains many tasks such as stereo, optical flow, visual\n",
      "    odometry, etc. This dataset contains the object detection dataset,\n",
      "    including the monocular images and bounding boxes. The dataset contains\n",
      "    7481 training images annotated with 3D bounding boxes. A full description\n",
      "    of the annotations can be found in the README of the object development kit\n",
      "    on the KITTI homepage.\n",
      "\n",
      "    Dataset size:\n",
      "        5.27 GiB\n",
      "\n",
      "    Source:\n",
      "        http://www.cvlibs.net/datasets/kitti\n",
      "    \n",
      "***** Supported splits *****\n",
      "test, train, validation\n",
      "\n",
      "***** Dataset location *****\n",
      "~/fiftyone/kitti\n",
      "\n",
      "***** Dataset info *****\n",
      "{\n",
      "    \"name\": \"kitti\",\n",
      "    \"zoo_dataset\": \"fiftyone.zoo.tf.KITTIDataset\",\n",
      "    \"dataset_type\": \"fiftyone.types.dataset_types.FiftyOneImageDetectionDataset\",\n",
      "    \"num_samples\": 423,\n",
      "    \"downloaded_splits\": {\n",
      "        \"validation\": {\n",
      "            \"split\": \"validation\",\n",
      "            \"num_samples\": 423\n",
      "        }\n",
      "    },\n",
      "    \"classes\": [\n",
      "        \"Car\",\n",
      "        \"Van\",\n",
      "        \"Truck\",\n",
      "        \"Pedestrian\",\n",
      "        \"Person_sitting\",\n",
      "        \"Cyclist\",\n",
      "        \"Tram\",\n",
      "        \"Misc\"\n",
      "    ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "fiftyone zoo info kitti"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The snippet below uses `fiftyone convert` to convert the test split of the CIFAR-10 dataset to [fiftyone.types.COCODetectionDataset](https://voxel51.com/docs/fiftyone/user_guide/export_datasets.html#cocodetectiondataset) format, which writes the dataset to disk with annotations in [COCO format](https://cocodataset.org/#home)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading dataset from '~/fiftyone/kitti/validation'\n",
      "Input format 'fiftyone.types.dataset_types.FiftyOneImageDetectionDataset'\n",
      " 100% |███████| 423/423 [1.2s elapsed, 0s remaining, 351.0 samples/s]         \n",
      "Import complete\n",
      "Exporting dataset to '/tmp/fiftyone/kitti-coco'\n",
      "Export format 'fiftyone.types.dataset_types.COCODetectionDataset'\n",
      " 100% |███████| 423/423 [4.4s elapsed, 0s remaining, 96.1 samples/s]       \n",
      "Export complete\n"
     ]
    }
   ],
   "source": [
    "INPUT_DIR=$(fiftyone zoo find kitti --split validation)\n",
    "OUTPUT_DIR=/tmp/fiftyone/kitti-coco\n",
    "\n",
    "fiftyone convert \\\n",
    "    --input-dir ${INPUT_DIR} --input-type fiftyone.types.FiftyOneImageDetectionDataset \\\n",
    "    --output-dir ${OUTPUT_DIR} --output-type fiftyone.types.COCODetectionDataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's verify that the conversion happened as expected:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 880\n",
      "drwxr-xr-x    4 voxel51  wheel   128B Jul 14 11:24 .\n",
      "drwxr-xr-x    5 voxel51  wheel   160B Jul 14 11:24 ..\n",
      "drwxr-xr-x  425 voxel51  wheel    13K Jul 14 11:24 data\n",
      "-rw-r--r--    1 voxel51  wheel   437K Jul 14 11:24 labels.json\n"
     ]
    }
   ],
   "source": [
    "ls -lah /tmp/fiftyone/kitti-coco/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 171008\n",
      "drwxr-xr-x  425 voxel51  wheel    13K Jul 14 11:24 .\n",
      "drwxr-xr-x    4 voxel51  wheel   128B Jul 14 11:24 ..\n",
      "-rw-r--r--    1 voxel51  wheel   195K Jul 14 11:24 000001.jpg\n",
      "-rw-r--r--    1 voxel51  wheel   191K Jul 14 11:24 000002.jpg\n",
      "-rw-r--r--    1 voxel51  wheel   167K Jul 14 11:24 000003.jpg\n",
      "-rw-r--r--    1 voxel51  wheel   196K Jul 14 11:24 000004.jpg\n",
      "-rw-r--r--    1 voxel51  wheel   224K Jul 14 11:24 000005.jpg\n",
      "-rw-r--r--    1 voxel51  wheel   195K Jul 14 11:24 000006.jpg\n",
      "-rw-r--r--    1 voxel51  wheel   177K Jul 14 11:24 000007.jpg\n"
     ]
    }
   ],
   "source": [
    "ls -lah /tmp/fiftyone/kitti-coco/data | head"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"info\": {\n",
      "        \"year\": \"\",\n",
      "        \"version\": \"\",\n",
      "        \"description\": \"Exported from FiftyOne\",\n",
      "        \"contributor\": \"\",\n",
      "        \"url\": \"https://voxel51.com/fiftyone\",\n",
      "        \"date_created\": \"2020-07-14T11:24:40\"\n",
      "    },\n",
      "    \"licenses\": [],\n",
      "    \"categories\": [\n",
      "        {\n",
      "            \"id\": 0,\n",
      "            \"name\": \"Car\",\n",
      "            \"supercategory\": \"none\"\n",
      "        },\n",
      "        {\n",
      "            \"id\": 1,\n",
      "            \"name\": \"Cyclist\",\n",
      "            \"supercategory\": \"none\"\n",
      "...\n",
      "            \"area\": 4545.8,\n",
      "            \"segmentation\": null,\n",
      "            \"iscrowd\": 0\n",
      "        },\n",
      "        {\n",
      "            \"id\": 3196,\n",
      "            \"image_id\": 422,\n",
      "            \"category_id\": 3,\n",
      "            \"bbox\": [\n",
      "                367.2,\n",
      "                107.3,\n",
      "                36.2,\n",
      "                105.2\n",
      "            ],\n",
      "            \"area\": 3808.2,\n",
      "            \"segmentation\": null,\n",
      "            \"iscrowd\": 0\n",
      "        }\n",
      "    ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "cat /tmp/fiftyone/kitti-coco/labels.json | python -m json.tool 2> /dev/null | head -20\n",
    "echo \"...\"\n",
    "cat /tmp/fiftyone/kitti-coco/labels.json | python -m json.tool 2> /dev/null | tail -20"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's convert from COCO format to [CVAT Image format](https://voxel51.com/docs/fiftyone/user_guide/export_datasets.html#cvatimageformat) format!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading dataset from '/tmp/fiftyone/kitti-coco'\n",
      "Input format 'fiftyone.types.dataset_types.COCODetectionDataset'\n",
      " 100% |███████| 423/423 [2.0s elapsed, 0s remaining, 206.4 samples/s]      \n",
      "Import complete\n",
      "Exporting dataset to '/tmp/fiftyone/kitti-cvat'\n",
      "Export format 'fiftyone.types.dataset_types.CVATImageDataset'\n",
      " 100% |███████| 423/423 [1.3s elapsed, 0s remaining, 323.7 samples/s]         \n",
      "Export complete\n"
     ]
    }
   ],
   "source": [
    "INPUT_DIR=/tmp/fiftyone/kitti-coco\n",
    "OUTPUT_DIR=/tmp/fiftyone/kitti-cvat\n",
    "\n",
    "fiftyone convert \\\n",
    "    --input-dir ${INPUT_DIR} --input-type fiftyone.types.COCODetectionDataset \\\n",
    "    --output-dir ${OUTPUT_DIR} --output-type fiftyone.types.CVATImageDataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's verify that the conversion happened as expected:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 584\n",
      "drwxr-xr-x    4 voxel51  wheel   128B Jul 14 11:25 .\n",
      "drwxr-xr-x    6 voxel51  wheel   192B Jul 14 11:25 ..\n",
      "drwxr-xr-x  425 voxel51  wheel    13K Jul 14 11:25 data\n",
      "-rw-r--r--    1 voxel51  wheel   289K Jul 14 11:25 labels.xml\n"
     ]
    }
   ],
   "source": [
    "ls -lah /tmp/fiftyone/kitti-cvat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
      "<annotations>\n",
      "    <version>1.1</version>\n",
      "    <meta>\n",
      "        <task>\n",
      "            <size>423</size>\n",
      "            <mode>annotation</mode>\n",
      "            <labels>\n",
      "                <label>\n",
      "                    <name>Car</name>\n",
      "                    <attributes>\n",
      "                    </attributes>\n",
      "                </label>\n",
      "                <label>\n",
      "                    <name>Cyclist</name>\n",
      "                    <attributes>\n",
      "                    </attributes>\n",
      "                </label>\n",
      "                <label>\n",
      "                    <name>Misc</name>\n",
      "...\n",
      "        <box label=\"Pedestrian\" xtl=\"360\" ytl=\"116\" xbr=\"402\" ybr=\"212\">\n",
      "        </box>\n",
      "        <box label=\"Pedestrian\" xtl=\"396\" ytl=\"120\" xbr=\"430\" ybr=\"212\">\n",
      "        </box>\n",
      "        <box label=\"Pedestrian\" xtl=\"413\" ytl=\"112\" xbr=\"483\" ybr=\"212\">\n",
      "        </box>\n",
      "        <box label=\"Pedestrian\" xtl=\"585\" ytl=\"80\" xbr=\"646\" ybr=\"215\">\n",
      "        </box>\n",
      "        <box label=\"Pedestrian\" xtl=\"635\" ytl=\"94\" xbr=\"688\" ybr=\"212\">\n",
      "        </box>\n",
      "        <box label=\"Pedestrian\" xtl=\"422\" ytl=\"85\" xbr=\"469\" ybr=\"210\">\n",
      "        </box>\n",
      "        <box label=\"Pedestrian\" xtl=\"457\" ytl=\"93\" xbr=\"520\" ybr=\"213\">\n",
      "        </box>\n",
      "        <box label=\"Pedestrian\" xtl=\"505\" ytl=\"101\" xbr=\"548\" ybr=\"206\">\n",
      "        </box>\n",
      "        <box label=\"Pedestrian\" xtl=\"367\" ytl=\"107\" xbr=\"403\" ybr=\"212\">\n",
      "        </box>\n",
      "    </image>\n",
      "</annotations>"
     ]
    }
   ],
   "source": [
    "cat /tmp/fiftyone/kitti-cvat/labels.xml | head -20\n",
    "echo \"...\"\n",
    "cat /tmp/fiftyone/kitti-cvat/labels.xml | tail -20"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleanup\n",
    "\n",
    "You can cleanup the files generated by this recipe by running the command below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "rm -rf /tmp/fiftyone"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
